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To  gauge  a  contractors  process  maturity  (on  individual  programs ),  the  Defense  Contract  Management  Agency  has 
applied  the  Software  Engineering  Institute 's  Capability  Maturity  Model®.  While  being  incrementally  deployed  this 
effort  is  already  paying  benefits  to  program  offices ,  contractors ,  and  the  Department  of  Defense.  The  goal  remains  con¬ 
tinuous  process  improvement  to  ensure  the  war  fighter,  the  end  user ;  receives  the  highest  quality  systems. 


Wouldn't  it  make  sense  to  have  a  way 
for  government  program  offices  to 
determine  the  maturity  of  a  contractor’s 
software  development  process  without 
incurring  the  cost  and  time  to  conduct  a 
total  software  capability  evaluation? 
Wouldn't  it  be  efficient  to  have  a  way  to 
eliminate  redundant  reviews  of  contractor 
software  development  processes  by  differ¬ 
ent  government  offices? 

Well,  now  there  is  a  way  to  obtain  this 
data.  Just  simply  ask.  While  not  fully  oper¬ 
ational  until  next  year,  the  capability  to 
provide  such  data  is  currently  in  place  at 
almost  half  of  the  field  locations  within 
the  Defense  Contract  Management 
Agency  (DCMA). 

As  the  on-site  government  representa¬ 
tives  at  contractor  facilities,  DCMA  pro¬ 
vides  assistance  to  all  branches  of  the  mili¬ 
tary.  Its  scope  of  effort  is  defined  within 
the  Federal  Acquisition  Regulation  (FAR). 
A  complete  description  of  DCMA  capa¬ 
bilities  was  previously  described  in 
Crosstalk  [1].  They  include  the  evalua¬ 
tion  and  surveillance  of  contractor  man¬ 
agement  systems  such  as  the  processes 
used  in  software  development  [2] .  For  this, 
the  agency  has  adopted  use  of  the  Software 
Engineering  Institute's  (SEI)  Capability 
Maturity  Model®  for  Software  (SW- 
CMM). 

The  SW-CMM  is  the  language  we 
needed  to  speak,  and  speak  fluently,  to 
communicate  with  the  broad  range  of  cus¬ 
tomers  across  the  Department  of  Defense 
(DoD).  It  is  the  language  spoken  by  gov¬ 
ernment  program  offices  when  conducting 
software  capability  evaluations  for  source 
selections  or  lesser  reviews.  It  is  the  lan¬ 
guage  selected  by  the  DoD  to  reduce  risk 
on  acquisitions  [3].  It  is  the  language 
employed  by  contractors  when  conducting 
a  CMM-based  appraisal  for  internal 
process  improvement  (CBA  IPI). 

®  The  Capability  Maturity  Model  and  CMM  are 
registered  in  the  U.S.  Patent  and  Trademark  Office. 


CMM-Based  Insight 

Our  initiative  to  speak  this  common  lan¬ 
guage,  what  we  call  CMM-based  insight, 
is  simple  in  concept.  Taking  advantage  of 
DCMA's  in-plant  presence,  we  will  prima¬ 
rily  organize  daily  observations  into  find¬ 
ings  based  on  the  CMM.  Observations 
undergo  an  internal  peer  review  for  con¬ 
formity  to  the  CMM,  then  data  is  freely 
shared  with  the  applicable  contractor  and 
passed  to  program  offices.  Findings  will  be 
used  to  concentrate  DCMA  effort  based 
on  risk.  Details  concerning  the  process, 
responsibilities,  and  outcomes  are  cap¬ 
tured  in  the  Method  Description 
Document  (MDD),  which  is  available  on 
line  at  www.dcma.mil/onebook/4.0/4.3/ini 
tiatives.htm. 

The  goals  (see  Table  1)  directly  benefit 
program  offices,  contractors,  and  the 
DoD.  Regardless  of  DCMA  location,  pro¬ 
gram  offices  will  have  consistent  data  con¬ 
cerning  a  contractor's  software  process 
maturity  for  programs  within  DCMA  cog¬ 
nizance.  Since  data  is  freely  shared  with 
the  contractor,  concern  or  disagreement 
on  high-risk  areas  can  be  resolved  at  the 
working  level,  or  elevated  as  necessary  to 
the  DCMA/Contractor/Program  Office 
Management  Council  [4] .  The  data  can  be 
used  in  future  process  reviews  to  reduce  or 
eliminate  redundant  areas.  The  results 
from  this  continuous  review  could  also  be 
used  as  a  vehicle  to  ensure  that  contractors 
have  maintained  a  process  capability  level 


CMM  Based  Insight  Goals 

1.  Provide  program  and  software  development  process 
risk  information  to  DCMA  and  buying  commands 

2.  Promote  supplier  process  improvements  based  on 
trend  analysis  of  CMM  based  observations 

3.  Consistently  maintain  data  to  identify  process 
capability  in  support  of  source  selection  and  contract 
monitoring 

4.  Promote  DCMA  internal  process  improvements 


Table  1 :  CMM-Based  Insight  Goals 

per  DoD  policy  [5]  or  in  support  of  inde¬ 
pendent  expert  program  reviews  of  soft¬ 
ware  intensive  systems  [6] . 

Evaluation  Relationships 

CMM-based  insight  is  not  a  software 
capability  evaluation  or  a  CBA  IPI  (see 
Table  2).  While  data  could  be  used  to  sub¬ 
stantiate  another  evaluation,  DCMA  will 
never  rate  a  particular  company  through 
CMM-based  insight.  The  initiative  is 
focused  on  identifying  areas  of  concern  on 
individual  programs  (i.e.,  higher  risk 
process  areas)  and  allocating  the  appropri¬ 
ate  level  of  resources  commensurate  with 
that  risk. 

Incremental  Phases 

As  previously  discussed,  the  initiative  is 
simple  in  concept.  But  like  the  process  of 
teaching  an  adult  to  speak  (and  think)  in  a 
new  language,  making  this  transition  has 


Table  2:  Comparison  of  Evaluation 


Software  Capability 
Evaluation 

CMM  -  Based  Appraisal 
for  Internal  Process 
Improvement 

DCMA  CMM  -  Based 
Insight 

Basis  of  Evaluation 

Software  CMM 

Software  CMM 

Software  CMM 

Company  Rating 
Provided 

No 

Yes 

No 

Frequency 

One  time 

One  Time 

Continuous 

Data  Refreshed 

No 

No 

Yes  (18  Month  Max) 

Conducting 

Organization 

Typically  government 
(including  DCMA) 

Contractor 

Government 

Basis  of  review 

Sponsor  selected,  usually 
within  a  particular 
domain 

Representative  programs 
across  a  business  base 

All  programs  within 
DCMA  cognizance 
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involved  a  culture  change  in  DCMA  soft¬ 
ware  surveillance  activities.  As  such,  incre¬ 
mental  phases  (see  Table  3)  were  designed 
to  assist  the  transition. 

Phase  I  validated  the  approach  at  the 
home  locations  of  our  agency  Software 
Engineering  Institute  (SEI)  affiliates. 
Phase  II  verified  that  approach  for  suit¬ 
ability  and  effectiveness  in  a  typical  field 
environment.  Phase  III  will  verify  the  cap¬ 
ture  and  transmission  of  data  before  the 
initiative  is  implemented  agency  wide. 

Data  Organization  Callenges 

The  primary  purpose  of  Phase  II  was  to 
verify  the  approach.  Due  to  the  shear 
number  of  inputs  -  necessary  for  the  cor¬ 
relation  of  observations  to  the  applicable 
key  practices,  internal  peer  reviews,  identi¬ 
fication,  and  subsequent  action  on  high 
risk  areas  -  the  need  for  an  adequate  sup¬ 
port  tool  was  recognized  early.  (This  situa¬ 
tion  will  be  resolved  in  Phase  III  when 
data  collection  is  incorporated  into  the 
common  tool  supporting  the  entire 
DCMA  Risk  Assessment  Management 
Program.)  Despite  this  burdensome  data 
collection,  currently  45  percent  of  our 
field  locations  have  volunteered  as  pilot 
locations  and  converted  operations.  Why? 
It  is  because  of  the  benefits  realized.  These 
are  perhaps  best  illustrated  with  actual 
examples. 

Example  1  -  Improvement  Not  Rating: 

A  program  office  concerned  with  a  history 
of  poor  software  quality  wanted  the  con¬ 
tractor  to  operate  at  CMM  Level  3.  The 
contracting  company’s  upper  manage¬ 


ment  believed  the  company  was  well  with¬ 
in  these  parameters  and  retained  an  out¬ 
side  consultant  to  verify  this  position. 
Initial  results  indicated  the  contractor  was 
operating  at  CMM  Level  3.  The  DCMA 
field  office  disagreed,  however,  based  upon 
observations  and  findings  per  the  CMM. 

Working  with  the  program  office,  the 
findings  were  questioned  and  the  issue  ele¬ 
vated  to  upper  management.  The  program 
office  held  that  if  the  review  revealed  a  sig¬ 
nificantly  different  result  than  that 
observed  in  day-to-day  operations,  the 
government  would  sponsor  an  independ¬ 
ent  software  capability  evaluation.  If  the 
government  evaluation  revealed  the  con¬ 
tractor  was  more  interested  in  paper  rat¬ 
ings  than  software  quality  improvement, 
the  government  would  consider  develop¬ 
ing  a  second  source  for  the  procurement. 

What  was  the  end  result?  The  final 
evaluation  revealed  operations  at  CMM 
Level  1.  The  contractor  was  well  on  the 
way  to  Level  2,  but  far  from  the  desired 
Level  3  target  profile.  Was  this  a  typical 
contractor  /  government  confrontation? 
Quite  the  opposite,  it  fostered  a  spirit  of 
process  improvement.  For  the  first  time, 
there  was  an  accurate  and  understood 
baseline.  The  contractor  developed  a 
roadmap  for  process  maturity  and  during 
the  course  of  two  years,  achieved  the 
desired  Level  3  profile.  DCMA,  the  gov¬ 
ernment  on-site  representative,  participat¬ 
ed  in  the  mini  reviews  and  was  a  team 
member  on  the  final  contractor-conduct¬ 
ed  CBA  IPI. 

Example  2  -  Risk  Rased  Operations:  If  a 

correlation  between  capability  (maturity 


level)  and  actual  performance  (cost,  sched¬ 
ule,  and  technical)  [7]  is  accepted,  it 
would  seem  reasonable  to  assume  that 
there  is  less  government  surveillance  of 
higher  maturity  operations  than  those 
with  lower  maturity.  In  the  absence  of 
data,  however,  people  often  focus  on  those 
areas  where  they  are  comfortable. 
Consequently,  a  low-risk  area  might  get  as 
much  attention  as  a  high-risk  area. 

This  is  not  so  with  the  CMM-based 
insight  methodology  because  it  is  based  on 
data  and  focuses  expended  effort  in  pro¬ 
portion  to  risk.  This  is  the  case  at  one  of 
our  pilot  locations  where  the  contractor 
has  achieved  CMM  Level  5.  The  CMM- 
based  insight  data  will  be  used  to  ensure 
that  DCMA  effort  and  resources  are  allo¬ 
cated  to  the  areas  of  highest  risk. 

Example  3  -  The  Rest  of  the  Story:  Is  a 

CMM  rating  truly  representative  of  all 
programs  at  a  given  facility?  As  the  name 
states,  the  model  measures  a  capability.  It 
would  seem  logical  to  assume  that  if  a 
capability  has  been  demonstrated  on  one 
program,  that  it  has  been  applied  to  all. 
With  mandated  levels,  though,  there  are 
other  pressures  that  come  into  play.  At  one 
pilot  location  the  contractor  had  conduct¬ 
ed  a  CBA  IPI  that  resulted  in  a  finding  of 
CMM  Level  3.  The  contractor  had  select¬ 
ed  programs  across  the  business  base  and 
then  hung  a  banner  over  the  building 
entrance  saying  "CMM  Level  3  Certified." 
What  was  wrong  with  that? 

First,  the  rating  Level  3  certified  is 
confusing  and  misleading.  Certified  by 
whom?  Secondly,  the  review  did  not 
include  the  largest  program,  one  which 
had  been  experiencing  problems  at  the 
international  level.  While  the  CBA  IPI 
shows  a  company’s  capability  to  operate  at 
a  given  level,  it  is  not  necessarily  true  for 
all  programs.  It  should  be,  and  seems  to  be 
in  most  cases,  especially  when  the  focus  is 
on  process  improvement.  However  in  this 
particular  case,  it  was  not.  With  the 
DCMA  data,  the  banner  was  removed  and 
the  applicable  program  office  understood 
that  operations  on  their  program  were  not 
at  CMM  Level  3,  and  why. 

Example  4  -  Eliminate/Reduce 

Duplicative  Reviews:  Concerned  about 
software  quality,  a  joint  program  office 
planned  a  review  of  the  contractor’s  soft¬ 
ware  development  processes.  The  DCMA 
pilot  location,  a  front-runner  for  this  ini¬ 
tiative,  already  had  the  data  in  the  com- 

July  2001 


Table  3:  CMM  Based  Insight  Implementation 


Start 

Phase 

Task 

Product 

Who 

New 

Sites 

Total 

Sites 

Oct  99 

I 

-Develop  &  Validate 
Approach 

-Method  Description 
Document  (MDD) 
-Training  Material 

DCMA  SEI 
Affiliates 

4 

4 

Jun  00 

IIA 

-Verify  Procedures 
-Validate  Data  Collection 
-Learn  &  employ 
methodology 

-Updated  MDD 
-Linalize  Data 
Requirements 

Volunteer  Eield 
Locations 

5 

9 

May  01 

IIB 

-Refine  approach 
-Learn  &  employ 
methodology 

-Procedure  Update 
(as  required) 

-Trained  personnel 

Volunteer  Eield 
Locations 

10 

19 

Aug  01 

IIC 

-Refine  approach 
-Learn  &  employ 
methodology 

-Procedure  Update 
(as  required) 

-Trained  personnel 

Volunteer  Field 
Locations 

5-10 

24-29 

TBD 

(Estimated 

Winter 

01/02) 

III 

Verify  Data  Integration  into 
DCMA  Risk  Assessment 
Management  Program 

-Data  collection  tool 
supporting  CMM 
based  operations 

All  Pilot  Sites 

0 

24-29 

TBD 

(Estimated 
Spring  02 

IV 

Agency  Wide  Deployment 

Process  Data  in 
terms  of  SW-CMM 

All  DCMA 
locations 

13-18 

42 
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mon  language  of  the  CMM.  It  clearly 
identified  strengths  and  weaknesses.  The 
review  was  cancelled  and  the  DCMA  data 
was  used  in  follow-on  actions  with  the 
contractor.  This  is  only  one  example,  but 
the  dollar  savings  across  the  department 
quickly  add  up.  The  cost  to  conduct  a 
software  capability  evaluation  has  been 
estimated  at  $50,000  for  both  the  govern¬ 
ment  and  contractor  [8] . 

Experience  and  Training 

A  complete  process  is  defined  as  having  1) 
procedures  and  methods  for  defining  the 
relationship  of  tasks,  2)  tools  and  equip¬ 
ment,  and  3)  people  with  skills,  training, 
and  motivation  [9] .  The  first  two  elements 
have  already  been  addressed.  Concerning 
people,  the  agency  has  more  than  400  per¬ 
sonnel  supporting  software  quality  assur¬ 
ance.  To  assure  this  workforce  is  properly 
prepared  to  deliver  consistently  first-rate 
assessments,  we  have  instituted  a  multi¬ 
phase  development  program: 

•  Basic  Training:  The  agency's  formal 
training  is  called  the  DCMA  Software 
Professional  Development  Program. 
Individuals  proceed  through  two  train¬ 
ing  levels.  Level  one  requires  complet¬ 
ing  72  hours  of  computer-based  train¬ 
ing,  40  hours  of  classroom  instruction, 
and  a  formal  mentoring  program 
focused  on  practical  application  of 
course  material.  Level  2  requires  an 
additional  97  hours  of  computer-based 
training,  120  hours  of  classroom 
instruction,  and  further  mentoring.  The 
SEI’s  CMM  is  integrated  into  the  com¬ 
puter-based  training,  classroom  instruc¬ 
tion,  and  mentoring.  Currently  78  per¬ 
cent  of  agency  software  personnel  have 
obtained  Level  2  status.  To  maintain 
this  level,  individuals  must  complete  a 
minimum  of  12  hours  of  software-relat¬ 
ed  training  each  year. 

•  Application  Training:  As  each  field  loca¬ 

tion  begins  operating  under  the  CMM- 
based  insight  initiative,  all  personnel 
undergo  an  additional  20  hours  of  spe¬ 
cific  application  training  conducted  on 
site  by  the  DCMA  Software  Center. 
Applicable  contractors  and  government 
program  offices  have  been  welcomed 
into  this  training.  It  is  specifically 
focused  on  the  initiative's  implementa¬ 
tion  and  daily  operation.  The  material 
was  developed  by  the  DCMA  Software 


Center  and  is  currently  being  reviewed 
by  the  SEI. 

•  On  Call  Assistance:  DCMA  personnel 
have  direct  access  to  the  six-person 
DCMA  Software  Center.  In  addition, 
one-eighth  of  the  total  field  workforce  has 
completed  the  SEI's  Software  Capability 
Evaluation  training.  Additional  assistance 
is  available  to  any  of  our  evaluators  from 
highly  qualified  agency  personnel  who 
are  SEI-certified  lead  assessors. 

•  Implementation  Measurement:  Training 
provides  a  foundation  for  conducting 
business  per  the  CMM,  but  it  does  not 
directly  correlate  to  experience,  which 
can  only  come  with  time.  Progress  in 
implementing  this  initiative  has  been 
promising.  For  instance,  more  and  more 
of  our  personnel  have  been  requested  as 
team  members  by  companies  when  con¬ 
ducting  CBA  IPIs.  However,  to  gauge 
implementation  progress  for  this  initia¬ 
tive  across  the  entire  agency  and  to 
make  necessary  adjustments,  the  agency 
is  employing  a  top-level  metric  based  on 
key  process  area  coverage.  Progress  will 
be  reviewed  by  the  agency  director,  his 
senior  leadership  team,  and  DCMA 
field  commanders. 

CMM-Based  Insight  and  CM  Ml 

The  baseline  for  our  efforts  is  the  SW- 
CMM.  We  fully  expect,  and  are  making 
preparations,  to  switch  over  to  the 
Capability  Maturity  Model  IntegratedSM 
(CMMISM  )  at  a  later  date.  The  agency  is 


part  of  the  SEI-led  CMMI  Steering  Group 
responsible  for  developing  the  SW- 
CMM/CMMI  turnover  within  the  DoD 
[10].  For  CMM-based  insight,  the  transi¬ 
tion  should  incur  little  breakage  moving 
to  the  integrated  model.  The  biggest  chal¬ 
lenge  in  using  either  model  is  the  disci¬ 
pline  and  knowledge  of  application  -  both 
of  which  we  are  gaining  with  our  current 
effort  and  are  fully  transferable.  Field  sites 
coming  aboard  in  each  phase  are  shown  in 
Table  4. 

DCMA  Creditability 

A  past  issue  of  CrossTalk  raised  the 
point,  "A  Level  3  development  effort  cou¬ 
pled  with  a  Level  1  acquiring  effort  often 
equates  to  a  Level  1  delivery  capability;  yet 
the  Level  3  developer  is  often  blamed,  and 
the  Software  (SW)  CMM  is  cited  as  inad¬ 
equate  [1  1] ."  I  saw  this  firsthand,  with  dis- 
asterous  results,  as  a  junior  officer.  So  how 
does  DCMA  measure  up? 

To  answer  that  question,  we  took  the 
sister  capability  maturity  model,  the 
Software  Acquisition  CMM,  and  tailored 
it  for  DCMA  use.  We  pilot  tested  and 
made  adjustments  as  applicable.  We  then 
went  agency  wide,  conducting  reviews 
from  November  1999  until  April  2000. 
Eight  equally  qualified  teams  were  used  to 
maintain  consistency.  What  were  the 
results?  There  were  a  few  organizations 
operating  at  the  defined  level  but  predom¬ 
inately  the  field  offices  within  the  agency 
operate  at  the  performed  level  (Level  1). 


Table  4:  Pilot  Locations 

DCMA  Pilot  Locations 

Phase  I  (Beginning  Oct  99) 

•  Boston,  Nashua 

•  Denver 

•  Delaware  Valley,  PA 

•  Syracuse 

Phase  II  (Added  Jun  00) 

•  Boeing,  Philadelphia 

•  St  Petersburg 

•  Lockheed  Martin  Owego,  N.Y. 

•  Sikorsky 

•  Lockheed  Martin  Sunnyvale,  CA 

Phase  II  B  (May  01) 

•  Birmingham,  Huntsville 

•  Hartford 

•  Bell  Helicopter,  Textron 

•  Baltimore,  Mannas sas 

•  Northrup  Grumman,  Bethpage 

•  Boeing,  St  Louis 

•  Northrup  Grumman,  Melbourne 

•  Orlando,  Harris,  Melbourne 

•  San  Antonio,  NASA,  Houston 

•  Springfield,  N .J. 

Phase  II  C  (Summer  01)  -  Up  to  10  Volunteer  Locations 

July  2001 
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More  importantly,  we  established  a 
solid  baseline  and  each  location  has  a 
detailed  roadmap  for  improvement  per  the 
model  structure.  Field  locations  have  been 
working  improvements  and  the  first  round 
of  follow-on  appraisals  began  in  the  spring 
of  2001.  The  original  evaluation  team 
members  constitute  the  personnel  pool  to 
support  independent  evaluation  of 
improvements  similar  to  the  industry 
approach  with  a  CBA  I  PI. 

Conclusion 

DCMA  was  always  required  and  continues 
to  conduct  evaluations  of  contractor's  soft¬ 
ware  development  processes  per  the  FAR. 
The  agency  is  now  deploying  a  standard 
methodology  via  continuous  process  eval¬ 
uations  that  is  organized  in  the  CMM,  the 
common  DoD  language,  and  is  based  on 
the  day-to-day  observations  of  the  in-plant 
DCMA  personnel.  Findings  are  peer- 
reviewed,  and  all  data  is  freely  shared  with 
the  applicable  contractor  and  is  available 
to  government  program  offices. 

While  full  agency  implementation  will 
not  occur  until  spring  2002,  the  approach 
has  been  developed  with  SEI  affiliates  and 
is  currently  in  pilot  testing  at  45  percent  of 
DCMA  field  locations.  Program  offices, 
the  contractors,  and  the  DoD  are  already 
realizing  benefits.  So,  how  much  does  the 
agency  believe  in  using  this  approach  to 
gauge  contractor  operations?  Enough  so 
that  we  are  walking  the  walk  and  measur¬ 
ing  our  operations  to  the  same  frame¬ 
work.  ♦ 
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